With the deprecation of Atlas Device Sync, Ditto is the recommended replacement for synchronizing edge devices with MongoDB. Refer to our Migration Guide for Atlas Device Sync for a comprehensive guide on moving from Atlas Device Sync to Ditto.
The MongoDB Connector is now generally available and is available to all paying Ditto customers.If you are interested in using the MongoDB connector, please visit the MongoDB Connector page, which has further details around requesting access.

​
How It Works

The MongoDB Connector is provided as a managed service, running alongside the Big Peer that you are using for your Ditto applications. The Big Peer is a centralized cloud datastore and synchronization engine that is analogous to the Atlas Device Sync service.
Only data currently stored within the Big Peer will be replicated to/from individual Small Peers (devices), so purpose of the connector is to synchronize data as it changes between MongoDB and the Big Peer. To do this, the connector listens to changes to documents in both Ditto and MongoDB. When it detects a change to a document it then performs conflict resolution between the two systems and updates the document in the target system with the new value. This conflict resolution mechanism roughly models Ditto’s causal consistency, ensuring that changes are not unexpectedly lost when mutations are made to the same documents in both MongoDB and Ditto simultaneously. Unlike other synchronization systems for MongoDB, Ditto’s advanced conflict resolution (based on CRDTs) avoids the need to deploy extra backend services to handle writes and conflicts, just deploy a Big Peer and you have everything you need to integrate with MongoDB.

​
Data Modelling Considerations

Both Ditto and MongoDB store their data in the form of JSON-like documents, so are a natural fit together when building applications that run on the edge. Unlike integrations with other systems that rely on synchronizing to relational forms (e.g. sqlite), you do not need to remodel your data to use Ditto and MongoDB together. However, there are a few considerations that you need to make when using Ditto.

​
ID Mapping Between Systems

Due to security considerations in a peer-to-peer context, Ditto’s permission system (see Data Authorization) only has access to the unique and immutable ID field (_id). To use permissions effectively, most Ditto applications use objects rather than strings as their _id. These sub-fields with the _id are then often duplicated in the document’s body to facilitate POJO/DTO-like patterns in your application code. For example you may choose to model an order (in a retail system) as follows:
JSON
{
    "_id": {
        "location": "LondonLiverpoolStreet",
        "orderId": "7c0c20ed-b285-48a6-80cd-6dcf06d52bcc"
    },
    "location": "LondonLiverpoolStreet",
    "orderId": "7c0c20ed-b285-48a6-80cd-6dcf06d52bcc",
    ...
}
This data model would allow you to effectively scope user’s permissions down to orders originating from a specific location, so that stores cannot view other stores orders. IDs in MongoDB, on the other hand, are commonly ObjectIds, essentially UUIDs with a time-based component; alternatively you may choose to use a single field value as the ID (e.g. orderId). It’s very rare for IDs in MongoDB to be used in the same way as within Ditto, therefore IDs likely need to be handled differently in both systems. The MongoDB Connector automatically bridges this difference in IDs, so that both systems can converge, even if their primary keys differ. It does this through the “ID Mapping” process, based on the configuration that you provide to the connector.
Fields used in the id mapping must be immutable and always present to achieve convergence between the two systems. If these requirements are not met, it will result in undefined behavior, such as duplicate Ditto documents with different IDs, or failure to synchronize the document between MongoDB and Ditto.
There are 3 modes for the ID mapping:
  • 1:1 ID (i.e. the same ID is used in both Ditto and MongoDB)
  • Single Document Field
  • Multiple Document Fields

​
1:1 ID

This mode is used if you select the “Match IDs” option, which will use the _id field in the ID mapping. In that case the ID fields used between Ditto and MongoDB will be identical. However, some restrictions apply here:
  • NULL cannot be present in the ID
  • Arrays cannot be present in the ID
  • The ID cannot be an object

​
Single Document Field

This mode is used if only a single field is specified in the field mapping.
You cannot use the _id field itself in this mode. You must only refer to top-level fields in the document, nested fields are not supported.
In this mode, all generated Ditto IDs will be set to the value of the selected field. For example, with a field mapping of just the orderId field and the following MongoDB document:
MongoDB
{
    "_id": ObjectId("abc123"),
    "location": "LondonLiverpoolStreet",
    "orderId": "7c0c20ed-b285-48a6-80cd-6dcf06d52bcc",
    "items": [
        "fries",
        "burger",
        "soda"
    ]
}
The connector would insert the following document into Ditto:
JSON
{
    "_id": "7c0c20ed-b285-48a6-80cd-6dcf06d52bcc",
    "location": "LondonLiverpoolStreet",
    "orderId": "7c0c20ed-b285-48a6-80cd-6dcf06d52bcc",
    "items": [
        "fries",
        "burger",
        "soda"
    ]
}

​
Multiple Document Fields

When the ID mapping contains multiple fields, like the restaurants collection in the example above, the Ditto ID will become an object with the specified fields and corresponding values from the document. Note that the Id Mapping cannot refer to the _id field itself in this case.
You cannot use the _id field itself in this mode. You must only refer to top-level fields in the document, nested fields are not supported.
For example, with a field mapping of the location and orderId fields and the following MongoDB document:
MongoDB
{
    "_id": ObjectId("abc123"),
    "location": "LondonLiverpoolStreet",
    "orderId": "7c0c20ed-b285-48a6-80cd-6dcf06d52bcc",
    "items": [
        "fries",
        "burger",
        "soda"
    ]
}
The connector would insert the following document into Ditto:
Ditto Document
{
    "_id": {
        "location": "LondonLiverpoolStreet",
        "orderId": "7c0c20ed-b285-48a6-80cd-6dcf06d52bcc"
    },
    "location": "LondonLiverpoolStreet",
    "orderId": "7c0c20ed-b285-48a6-80cd-6dcf06d52bcc",
    "items": [
        "fries",
        "burger",
        "soda"
    ]
}

​
MongoDB Datatypes

MongoDB stores data as BSON, which includes rich datatypes such as native date objects, you can read more about the datatypes available in https://www.mongodb.com/docs/manual/reference/bson-types/. Ditto represents its data using JSON with a few key additional datatypes (e.g. integer), which does not have the full set of datatypes available in BSON. The connector will perform automatic mapping of BSON datatypes to JSON when synchronizing data from MongoDB to Ditto. The full set of mappings are listed below:
MongoDB BSON DatatypeDitto JSON Datatype
Doublenumber
Stringstring
Objectobject
Arrayarray
ObjectIdstring
Booleanboolean
Datestring (ISO8601 format)
Nullnull
Regular Expressionobject (see EJSON v2)
32-bit Integerinteger
64-bit Integerinteger
Timestampnumber (epoch timestamp)
Decimal128string
All BSON datatypes that are not listed above will not be replicated between Ditto and MongoDB.
When replicating data from MongoDB to Ditto, the connector will store metadata allowing the connector to retain the mapping between datatypes when writing the same document back to MongoDB. In the case that a document is created within Ditto, the document will be inserted into MongoDB using only the basic JSON datatypes.

​
Setting Up the Connector

To setup the connector, you first must carry out some steps within your MongoDB cluster, then configure the connector via API to launch the connector.

​
Pre-requisites

You need to have a MongoDB cluster already setup, the minimum supported version for the connector is currently MongoDB 7.0.13 and MongoDB 8.0.0. You need to follow a number of steps to ensure that MongoDB is ready to be used with the connector.

​
Create MongoDB Database

If your target database doesn’t currently exist, then you must first create this within MongoDB. No special configuration is required for the created database. An existing database can be used if one exists.

​
Create MongoDB Collections

All collections you wish to synchronize with Ditto must exist within MongoDB before setting up the connector. Each of these synchronized collections must have changeStreamPreAndPostImages set to true. This setting allows the connector to ensure causal consistency between both systems, and the connector will not start if this is not set for all collections. When creating new collections, pass changeStreamPreAndPostImages: true to your creation command. For pre-existing collections, you can modify this configuration using collMod. See Enable Change Stream Pre and Post Images. For example, to edit an existing collection called orders, you can run the following command: 
db.runCommand({ collMod: "orders", changeStreamPreAndPostImages: {"enabled": true} })
If you are using pre-existing collections, existing data in these collections will not be synchronized unless they are modified after the connector has been setup.

​
Create a MongoDB Database User

The connector authenticates to MongoDB using the username and password of a database user. We recommend creating a dedicated database user for the connector to use within MongoDB. At a minimum the database user will require the readWrite permission for the specific database that you’re looking to synchronize with Ditto. See Configure MongoDB Database Users. If you are connecting to a sharded database (i.e. a mongos connection) the database user also needs to have the explicit read permission for the built-in collection collection in the config database, so it can inspect sharding configurations.

​
Add Ditto IPs to MongoDB Allowlist

The Big Peer uses the following 3 IPs for interacting with external services such as MongoDB: 
3.147.233.88
52.15.232.117
3.130.255.9
You need to add these three IPs to the allowlist of your MongoDB cluster to allow the connector to access the cluster correctly, see Add IPs to MongoDB Allowlist.

​
Configuring the Connector

To get access to the MongoDB Connector UI, you’ll need to contact Ditto to have your organization enrolled.
The connector is configured using the Ditto Portal. You can find the MongoDB Connector UI in the Ditto Portal, under Settings > MongoDB Connector.

​
Step-by-Step Guide

1

Fill in the database field, with the MongoDB database that you wish to synchronize with Ditto.
2

Fill in the connection string field, with the connection string of the MongoDB database that you wish to synchronize with Ditto. You can find this connection string in the MongoDB Atlas UI, under the Connect button for your cluster, ensuring that you use the credentials of the database user that you created in the prerequisites.The connection string must be supplied in the format mongodb+srv://<username>:<password>@<cluster>.mongodb.net/. For example to connect to the cluster with the URL my_cluster.mongodb.net, using the username my_user with a password of my_password, you would provide mongodb+srv://my_user:my_password@my_cluster.mongodb.net/. Non-SRV-based connection strings cannot be used with the MongoDB connector.
While the password is passed into the Ditto Portal as part of the connection string, it is stored securely in Ditto’s systems and is not accessible to Ditto staff.
3

Fill in the ID mapping field, with the ID mapping that you wish to use for the connector.You can add new collections to the connector by clicking the Add Collection button, entering the collection name, toggling initial sync, then selecting how you would like your IDs mapped.Enabling initial sync will trigger an initial sync of the collection from MongoDB to Ditto, this is rate limited and will be done in the background to avoid affecting the performance of the connector. Once the initial sync is complete, the connector will continue to synchronize the collection on an ongoing basis via MongoDB Change Streams.For very large collections, you may opt to not rely on the Connector to perform the initial sync, and instead perform the initial sync manually via the Ditto API so that you have finer-grained control over the process.If you select the Match IDs option, then the ID fields used between Ditto and MongoDB will be identical. If you select the Map fields to Ditto ID option, then you will need to add all of the fields that you wish to map to the Ditto ID. Once done, click the Add collection button to add the collection to the connector configuration, repeat this step for each collection you wish to synchronize with Ditto.
ID Mapping is a critical part of configuring the connector, please makes sure you read the ID Mapping section carefully and configure as necessary for your use case.
4

Once all of the fields have been added, click the Save button to create the connector.
5

Once the connector has been created, you will be able to see the status of the connector in the Ditto Portal. It will first show as Pending while the connector is being created, then once it is ready it will show as Running. In case of an error, the connector will show as Failed, and you will be able to see the error message in the Ditto Portal, see also Troubleshooting Connectivity.

​
Troubleshooting Connectivity

When the connector starts up, it will perform a number of checks to ensure that it is able to connect to MongoDB and that the configuration is correct. If any of these checks fail, the connector will show as Failed, and you will be able to see a more detailed error message in the Ditto Portal. These checks are as follows:
  • Ensure that the MongoDB connection string is reachable
  • Ensure that the MongoDB database user has the correct permissions
  • Ensure that the MongoDB cluster has the correct IP addresses added to the allowlist
  • Ensure that the MongoDB database exists
  • Ensure that the MongoDB collections exist
  • Ensure that the MongoDB collections have changeStreamPreAndPostImages set to true
  • Ensure that it can create all of the necessary metadata collections
Below are the error messages that you may encounter, and the remediation steps you can take to resolve them. Remember that if you need to modify the configuration of a running connector, you must delete the connector and create it again with the new configuration.
Error MessageRemediation
Connection string provided is invalid.Delete the connector and try again, ensuring that the connection string is correct.
Failed to connect to MongoDB• Ensure that the MongoDB cluster has the correct IP addresses added to the allowlist (see Add Ditto IPs to MongoDB Allowlist)
• Ensure that the MongoDB database user exists and has the correct permissions (see Create a MongoDB Database User)
Failed to list database namesEnsure that the MongoDB database user has the correct permissions (see Create a MongoDB Database User)
Failed to list collectionsEnsure that the MongoDB database user has the correct permissions (see Create a MongoDB Database User)
Database <name> not foundEnsure that the MongoDB database exists (see Create MongoDB Database)
Collection <name> not foundEnsure that the MongoDB collection exists (see Create MongoDB Collections)
Collection <name> must be configured with changeStreamPreAndPostImages enabledEnsure that the MongoDB collection has changeStreamPreAndPostImages set to true (see Create MongoDB Collections)
Collection <name> cannot be a capped collectionEnsure that the MongoDB collection is not a capped collection (see Create MongoDB Collections)
Collection <name> cannot be createdEnsure that the MongoDB database user has the correct permissions (see Create a MongoDB Database User)
Something went wrongAn unknown error occurred, please contact Ditto support for assistance.

​
Reconfiguring the Connector

There are some cases where you may need to re-configure the connector, for example:
  • You need to add a new collection to the connector
  • You need to remove a collection from the connector
  • You need to change the ID mapping for a collection
  • You need to change the credentials used by the connector
The connector supports re-configuration, so you can add new collections, remove collections, change the ID mapping, and change the credentials used by the connector while the connector is running. You can re-configure the connector by clicking the Edit button on the MongoDB Connector page in the Ditto Portal. This will open a form with the current configuration, allowing you to modify the configuration as required: You can make any changes to the configuration that you would like, then click the Save button to update the connector with the new configuration.

​
Behavior on Re-configuration

When you re-configure the connector, the connector will automatically take specific actions based on the changes you have made:
  • If adding a new collection, the connector will automatically trigger an initial sync of the collection from MongoDB to Ditto, if initial sync is enabled in the collection configuration.
  • If changing the ID mapping for a collection, the connector will automatically trigger an initial sync of the collection from MongoDB to Ditto if enabled in the collection configuration. All documents imported during this initial sync will be using the new ID mapping, but existing documents in Ditto will not be updated, you will need to manually delete these old documents in Ditto if desired.
  • If toggling initial sync on a collection, the connector will not automatically trigger an initial sync of the collection from MongoDB to Ditto, unless there are other changes to the collection configuration (e.g. ID mapping).
  • If removing a collection, the connector will stop synchronizing the collection between MongoDB and Ditto. Existing documents in this Ditto collection will not be deleted you will need to manually delete these documents in Ditto if desired.
  • If changing the credentials used by the connector, the connector will automatically reconnect to MongoDB using the new credentials. No document changes will be missed as part of this rotation process.

​
Internal Metadata

In order to provide consistency across the two systems, the connector uses some internal metadata. Specifically this metadata tracks:
  • Current MongoDB sessions, to avoid circular writes between the two systems. This is stored in the __ditto_connector_sessions internal collection within your MongoDB database. These are periodically cleaned up by the connector when the sessions are no longer needed.
  • Documents that failed to synchronize from Ditto to MongoDB (for example they may have been rejected due to schema validation issues). These are stored in the __ditto_unsynced_documents internal collection within your MongoDB database. These documents serve as a record for you to know which documents have failed to synchronize, and can be used to manually resync the documents. This collection is not automatically cleaned up by the connector, you can choose to delete the documents in this collection if you no longer need to keep track of these documents.
  • BSON field mappings for documents, currently stored as a metadata field (prefixed with _ within the Ditto document)

​
Interaction with Deletes

As the document within Ditto contains the metadata, when a document is deleted, this metadata is also deleted. To provide sensible convergence behavior between the two systems, Ditto will treat any deletions coming from MongoDB as “winning” over any changes made to the document in Ditto. This is even true if the document is later recreated on a device within Ditto. If the document is later recreated in MongoDB, the connector will detect this and will update the document in Ditto to reflect the new state from MongoDB. You should generally treat deletions as a final state, and not as a reversible state, if you need to then you should do this via MongoDB rather than via Ditto. To have utmost control over the behavior of deletions, you can follow a soft-delete pattern as recommended in Accessing Data.